The present applicant has already proposed class classification adaptive processing as a method for improving the quality of images and that of sound.
The class classification adaptive processing is formed of class classification processing and adaptive processing. Data is classified into classes according to its nature in the class classification processing, and the adaptive processing is applied to each class. The adaptive processing is performed in the following way.
Specifically, for example, assuming now that images are to be handled, in the adaptive processing, for example, pixel values of a low-quality image are linearly coupled with predetermined tap coefficients to obtain prediction values of pixel values of a high-quality image, so that the low-quality image is converted to the high-quality image.
More specifically, for example, it is assumed now that a high-quality image is set to master data; a decoded image having a reduced image quality, obtained by encoding the high-quality image, for example, by a JPEG (Joint Photographic Experts Group) method or an MPEG (Moving Picture Experts Group) method, and then decoding encoded data, is set to apprentice data; and a prediction value E[y] of a high-quality pixel value “y,” serving as maser data, is obtained by a linear coupling model defined by linear coupling of a set of low-quality pixel values, x1, x2, . . . , serving as apprentice data, and predetermined tap coefficients w1, w2, . . . . In this case, the prediction value E[y] can be expressed by the following expression.E[y]=w1x1+w2x2+ . . .  (1)
To generalize expression (1), when a matrix “W” formed of a set of tap coefficients “wj,” a matrix “X” formed of a set of apprentice data “xij,” and a matrix “Y′” formed of a set of prediction values E[yj] are defined in the following expressions,
                              X          =                      [                                                                                x                    11                                                                                        x                    12                                                                    ⋯                                                                      x                                          1                      ⁢                      J                                                                                                                                        x                    21                                                                                        x                    22                                                                    ⋯                                                                      x                                          2                      ⁢                      J                                                                                                                    ⋯                                                  ⋯                                                  ⋯                                                  ⋯                                                                                                  x                                          I                      ⁢                                                                                          ⁢                      1                                                                                                            x                                          I                      ⁢                                                                                          ⁢                      2                                                                                        ⋯                                                                      x                    IJ                                                                        ]                          ⁢                                  ⁢                              W            =                          [                                                                                          W                      1                                                                                                                                  W                      2                                                                                                            ⋯                                                                                                              W                      J                                                                                  ]                                ,                                    Y              ′                        =                          [                                                                                          E                      ⁡                                              [                                                  y                          1                                                ]                                                                                                                                                        E                      ⁡                                              [                                                  y                          2                                                ]                                                                                                                                  ⋯                                                                                                              E                      ⁡                                              [                                                  y                          I                                                ]                                                                                                        ]                                                          [                  Expression          ⁢                                          ⁢          1                ]            the following observation equation is satisfied.XW=Y′  (2)where, an element xij in the matrix X indicates j-th apprentice data in i-th set (set of apprentice data used for predicting i-th master data yi) of apprentice data, an element wj in the matrix W indicates a tap coefficient to be multiplied by the j-th apprentice data in a set of apprentice data, and yi indicates i-th master data. Therefore, E[yi] indicates a prediction value for the i-th master data. In expression (1), the suffix i in the element yi of the matrix Y is omitted to show y at the left-hand side, and the suffix i in the element xij of the matrix X is also omitted to show x1, x2, . . . at the right-hand side.
It is assumed that the least squares method is applied to this observation equation (2) to obtain a prediction value E[y] close to the high-quality pixel value “y.” In this case, when a matrix “Y” formed of a set of the pixel values “y” serving as master data, and a matrix “E” formed of a set of the remainders “e” of the prediction values E[y] with respect to the pixel values “y” are defined in the following way,
                              E          =                      [                                                                                e                    1                                                                                                                    e                    2                                                                                                ⋯                                                                                                  e                    T                                                                        ]                          ,                  Y          =                      [                                                                                y                    1                                                                                                                    y                    2                                                                                                ⋯                                                                                                  y                    T                                                                        ]                                              [                  Expression          ⁢                                          ⁢          2                ]            the following remainder equation is satisfied from expression (2).XW=Y+E  (3)
In this case, tap coefficients wi used to obtain a prediction value E[y] close to the original pixel value “y” are obtained when the following square error is made minimum.
                              ∑                      i            =            1                    I                ⁢                  e          i          2                                    [                  Expression          ⁢                                          ⁢          3                ]            
Therefore, when the result obtained by differentiating the above square error by the prediction coefficients wj is zero, in other words, the prediction coefficients wj which satisfy the following expression are the most appropriate values for obtaining a prediction value E[y] close to the pixel value “y.”
      [          Expression      ⁢                          ⁢      4        ]                                                                          e                1                            ⁢                                                ∂                                      e                    1                                                                    ∂                                      w                    j                                                                        +                                          e                2                            ⁢                                                ∂                                      e                    2                                                                    ∂                                      w                    j                                                                        +            ⋯            +                                          e                I                            ⁢                                                ∂                                      e                    I                                                                    ∂                                      w                    j                                                                                =                      0            ⁢                          (                                                j                  =                  1                                ,                2                ,                ⋯                ⁢                                                                  ,                                                                  ⁢                J                            )                                                            (          4          )                    
When expression (3) is differentiated by the prediction coefficients wj, the following expression is satisfied.
      [          Expression      ⁢                          ⁢      5        ]                                                                          ∂                                  e                  i                                                            ∂                                  w                  1                                                      =                          x                              i                ⁢                                                                  ⁢                1                                              ,                                                    ∂                                  e                  i                                                            ∂                                  w                  2                                                      =                          x                              i                ⁢                                                                  ⁢                2                                              ,          ⋯          ⁢                                          ,                                          ⁢                                                    ∂                                  e                  i                                                            ∂                                  w                  J                                                      =                          x                              i                ⁢                                                                  ⁢                J                                              ,                      (                                          i                =                1                            ,              2              ,              ⋯              ⁢                                                          ,                                                          ⁢              I                        )                                                (          5          )                    
From expressions (4) and (5), expression (6) is obtained.
      [          Expression      ⁢                          ⁢      6        ]                                                                          ∑                                  i                  =                  1                                I                            ⁢                                                e                  i                                ⁢                                  x                                      i                    ⁢                                                                                  ⁢                    1                                                                        =            0                    ,                                                    ∑                                  i                  =                  1                                I                            ⁢                                                e                  i                                ⁢                                  x                                      i                    ⁢                                                                                  ⁢                    2                                                                        =            0                    ,                                    ⋯              ⁢                                                          ⁢                                                ∑                                      i                    =                    1                                    I                                ⁢                                                      e                    i                                    ⁢                                      x                                          i                      ⁢                                                                                          ⁢                      J                                                                                            =            0.                                                (          6          )                    
In addition, when the relationships among the apprentice data “xij,” the tap coefficients “wj,” the master data “yi,” and the remainders “ei” in the remainder equation (3) are taken into account, the following normal equations are obtained from expression (6).
      [          Expression      ⁢                          ⁢      7        ]                                {                                                                                                                                        (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                1                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                1                                                                                                                                    )                                            ⁢                                              W                        1                                                              +                                                                  (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                1                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                2                                                                                                                                    )                                            ⁢                                              W                        2                                                              +                    ⋯                    ⁢                                                                                  +                                                                  (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                1                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                J                                                                                                                                    )                                            ⁢                                              W                        J                                                                              =                                      (                                                                  ∑                                                  i                          =                          1                                                I                                            ⁢                                                                        x                                                      i                            ⁢                                                                                                                  ⁢                            1                                                                          ⁢                                                  y                          i                                                                                      )                                                                                                                                                                                          (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                2                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                1                                                                                                                                    )                                            ⁢                                              W                        1                                                              +                                                                  (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                2                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                2                                                                                                                                    )                                            ⁢                                              W                        2                                                              +                    ⋯                    ⁢                                                                                  +                                                                  (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                2                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                J                                                                                                                                    )                                            ⁢                                              W                        J                                                                              =                                      (                                                                  ∑                                                  i                          =                          1                                                I                                            ⁢                                                                        x                                                      i                            ⁢                                                                                                                  ⁢                            2                                                                          ⁢                                                  y                          i                                                                                      )                                                                                                                                                                                          (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                J                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                1                                                                                                                                    )                                            ⁢                                              W                        1                                                              +                                                                  (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                J                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                2                                                                                                                                    )                                            ⁢                                              W                        2                                                              +                    ⋯                    ⁢                                                                                  +                                                                  (                                                                              ∑                                                          i                              =                              1                                                        I                                                    ⁢                                                                                    x                                                              i                                ⁢                                                                                                                                  ⁢                                J                                                                                      ⁢                                                          x                                                              i                                ⁢                                                                                                                                  ⁢                                J                                                                                                                                    )                                            ⁢                                              W                        J                                                                              =                                      (                                                                  ∑                                                  i                          =                          1                                                I                                            ⁢                                                                        x                                                      i                            ⁢                                                                                                                  ⁢                            J                                                                          ⁢                                                  y                          i                                                                                      )                                                                                                            (          7          )                    
When a matrix (covariance matrix) A and a vector “v” are defined in the following way,
                              A          =                      (                                                                                                      ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                                                                                                                                ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                                                                                      ⋯                                                                                            ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                                                                                                                                                            ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                                                                                                                                ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                                                                                      ⋯                                                                                            ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                                                                                                                                                                                                                                                                                                  ⋯                                                                                                                                                                                                                ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                                                                                                                                ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                                                                                      ⋯                                                                                            ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                    ⁢                                              x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                                                                                          )                          ⁢                                  ⁢                  v          =                      (                                                                                                      ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          1                                                                    ⁢                                              y                        i                                                                                                                                                                                    ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          2                                                                    ⁢                                              y                        i                                                                                                                                          ⋮                                                                                                                        ∑                                              i                        =                        1                                            I                                        ⁢                                                                  x                                                  i                          ⁢                                                                                                          ⁢                          J                                                                    ⁢                                              y                        i                                                                                                                  )                                              [                  Expression          ⁢                                          ⁢          8                ]            and the vector W is defined as shown in expression (1), the normal equations shown in (7) can be expressed in the following way.AW=v  (8)
The same number of normal equations (7) as that “J” of the tap coefficients “wj” to be obtained can be formulated if a certain number of sets of apprentice data xij and master data yi are prepared. Therefore, by solving equation (8) for the vector W (to solve equation (8), the matrix A in equation (8) needs to be a non-singular matrix), the tap coefficients “wj” which make a statistical prediction error minimum (in this case, which make the square error minimum) is obtained. It is possible to use, for example, a sweep off method (Gauss-Jordan deletion method) to solve equation (8).
As described above, in the adaptive processing, the tap coefficients wj which make the statistical prediction error minimum are obtained, and then, a prediction value E[y] close to the high-quality pixel value “y” is obtained from expression (1) by the use of the tap coefficients wj.
When an image having the same image quality as the image to be encoded is used as master data, and a decoded image obtained by first encoding the master data and then decoding it is used as apprentice data, for example, tap coefficients which make the statistical prediction error minimum in a process of decoding the encoded image data to generate the original image data are obtained.
When an image having a higher resolution than the image to be encoded is used as master data, and a decoded image obtained by reducing the resolution of the master data to the same image quality as the image to be encoded, encoding it, and then decoding it is used as apprentice data, for example, tap coefficients which make the statistical prediction error minimum in a process of decoding the encoded image data to generate the high-resolution image data are obtained.
Therefore, in this case, according to the adaptive processing, depending on selecting an image used for master data or apprentice data, tap coefficients are obtained which can adjust the image quality of a decoded image as desired.
The adaptive processing largely differs from usual filtering implemented by a filter in that tap coefficients which make the statistical prediction error minimum by learning are used, and a high-frequency component not included in the image to be encoded is reproduced.
Therefore, when class classification adaptive processing is applied to a decoded image obtained by decoding encoded data, the decoded image has an improved image quality.
If the image quality of a decoded image obtained by decoding encoded data is very low due to, for example, the state of a transmission path through which the encoded data is transmitted, and a compression rate at which image data was encoded, however, even when class classification adaptive processing is executed only by the use of the decoded image, it is difficult to sufficiently improve its image quality in some cases.